Surgical devices and methods of use thereof

ABSTRACT

The present invention provides a method, including: obtaining a first image from a first imaging modality; identifying on the first image from the first imaging modality obtaining a second image from a second imaging modality; generating a compatible virtual image from the first image from the first imaging modality; mapping planning data on the compatible virtual image; coarse registering of the second image from the second imaging modality to the first image from the first imaging modality; identifying at least one element of the mapped planning data from the compatible virtual image; identifying at least one corresponding element on the second imaging modality; mapping the at least one corresponding element on the second imaging modality; fine registering of the second image from the second imaging modality to the first image from the first imaging modality; generating a third image.

RELATED APPLICATIONS

This application claims the priority of U.S. provisional applicationSer. No. 61/923,956, entitled “AUGMENTED FLUOROSCOPY,” filed Jan. 6,2014, U.S. provisional application Ser. No. 62/013,726, entitled“AUGMENTED FLUOROSCOPY,” filed Jun. 18, 2014, and U.S. provisionalapplication Ser. No. 62/052,039, entitled “AUGMENTED FLUOROSCOPY,” filedSep. 18, 2014, which are incorporated herein by reference in theirentireties for all purposes.

FIELD OF THE INVENTION

The embodiments of the present invention relate to surgical devices andmethods of use thereof.

BACKGROUND OF INVENTION

Use of video-assisted thoracic surgery (VATS) during endoscopic surgery,as well as other fields of surgery, can be used during the treatment ofvarious respiratory diseases.

BRIEF SUMMARY OF INVENTION

In some embodiments, the instant invention provides a method, including:obtaining a first image from a first imaging modality; identifying onthe first image from the first imaging modality at least one element,where the at least one element comprises a landmark, an area ofinterest, an incision point, a bifurcation, an organ, or any combinationthereof, obtaining a second image from a second imaging modality;generating a compatible virtual image from the first image from thefirst imaging modality; mapping planning data on the compatible virtualimage; where mapped planning data corresponds to the at least oneelement, coarse registering of the second image from the second imagingmodality to the first image from the first imaging modality; identifyingat least one element of the mapped planning data from the compatiblevirtual image; identifying at least one corresponding element on thesecond imaging modality; mapping the at least one corresponding elementon the second imaging modality; fine registering of the second imagefrom the second imaging modality to the first image from the firstimaging modality; generating a third image; where the third image is anaugmented image including a highlighted area of interest.

In some embodiments, the method further includes superimposing the atleast one image, a portion of the at least one image, or a planninginformation derived from the first imaging modality over the secondimaging modality. In some embodiments, the method further includes usingat least one instruction, where the at least one instruction can includeinformation regarding navigation, guidance, or a combination thereof. Insome embodiments, the guidance includes information regarding apositioning of a device shown the second imaging modality, where thedevice comprises a fluoroscopic C-Arm, as to result in achievingvisibility for the area of interest, incision points, anatomicalstructures, or tool access direction. In some embodiments, the methodfurther includes tracking of at least one anatomical structure by use ofat least one subsequent image derived from the second imaging modality,where the second imaging modality comprises a fluoroscopic videoconfigured to have substantially the same acquisition parameters, andwhere the acquisition parameters comprise mode, position, field of view,or any combination thereof, to generate the augmented fluoroscopic imageby suppressing static anatomic structures and/or improving signal tonoise of underlying soft tissue. In some embodiments, the method furtherincludes performing a multiphase registration, where the at least onesubstantially static object is first registered; and where at least onedynamic object is second registered, where the at least one dynamicobject comprises a diaphragm, a bronchus, a blood vessel, or anycombination thereof. In some embodiments, the method further includesdeemphasizing at least one interfering structure. In some embodiments,the compatible virtual image is not generated while the planning datafrom first imaging modality is transferred to second imaging modality bymeans of image registration.

In some embodiments, the instant invention provides a method, including:

using at least two intraoperative images with known relative movementand rotation to generate a grouping of pixels derived from anintraoperative image, where the grouping of pixels is determined byindividual calculation of each pixel using: (a) movement variation ofeach pixel and (b) intensity values of each pixel; performingregistration using at least two sequential intraoperative images toreconstruct structures in an area of interest; differentiating movingstructures from static structures in the area of interest; andhighlighting anatomical structures on at least one intraoperative image.In some embodiments, the method further includes using a chest x-rayradiographic image as a first intraoperative image.

In some embodiments, the instant invention provides a system includingan augmented fluoroscopy device configured to generate an augmentedfluoroscopy image including (a) video and image processing unit, (b)video input card or externally connected device configured to inputvideo signal a fluoroscopic device, (c) 3D planning input in internal orDICOM format, (d) an augmented video signal output, or any combinationthereof. In some embodiments, the system is integrated with at least onefluoroscopic device is a module including a RAW data input card (i.e.,instead of a video input card) configured to obtain RAW data as asignal. In some embodiments, the system is integrated with a Cone-beamCT system.

In some embodiments, the instant invention provides a system includingan instrument for navigating inside natural body cavity including: (a) aguided sheath with anchoring at the tip and/or (b) a guided wire. Insome embodiments, the instrument is an inflatable balloon configured toact as an anchoring mechanism.

In some embodiments, the instant invention provides a method including:(i) selecting a volume of interest on a first image from a first imagingmodality; (ii) generating a second image from a second imaging modality;(iii) coarse registering using the first imaging modality and the secondimaging modality; (iv) producing at least one pattern from the firstimaging modality; (v) generating a matching pattern by use of the secondimaging modality using single or multiple patterns produced from firstimaging modality; (vi) enhancing the matching pattern from the secondimaging modality to highlight the anatomy in the volume of interest forproducing third imaging modality. In some embodiments, the anatomicstructures located outside the area of interest are found and suppressedusing substantially the same method. In some embodiments, the patternincludes anatomical features including, but not limited to, airways,ribs, and blood vessels. In some embodiments, the matching feature fromsecond imaging modality is derived from a set of at least one instrumentposition inside the area of interest.

In some embodiments, the instant invention provides a method including:using a first imaging modality to obtain at least one first image of apatient's chest; segmenting natural body cavities including bronchialairways in a 3D space; generating at least one image from a secondimaging modality; generating a two-dimensional augmented image generatedfrom the second imaging modality by combining information, where theinformation describes a complete map or a partial map of natural bodycavities, including a bronchial airway tree; calculating registrationbetween the first imaging modality and the second imaging modality aspose estimation between the portion of bronchial airway sourcing fromsecond imaging modality and segmented map of bronchial airway sourcingfrom first imaging modality; calculating registration between first andsecond imaging modalities through pose estimation by mappingcorresponding features. In some embodiments, the augmented bronchogramis generated using radiopaque material is injected to highlight the bodycavity. In some embodiments, the augmented bronchogram is generatedthrough superposition of imaging from at least three two differentpositions of radiopaque instrument located inside the body cavities. Insome embodiments, an augmented bronchogram is generated throughsuperposition of imaging from at least one different positions ofradiopaque instrument located inside the body cavity and angularmeasurement of C-Arm orientation relative to patient bed. In someembodiments, the radiopaque instrument is designed and configured toreconstruct its three-dimensional space from single projection. In someembodiments, the radiopaque substance(s) having a high viscosity suchas, but not limited to, hydrogel, reverse thermo-gelling polymer can beused to generate augmented bronchogram.

In some embodiments, the instant invention provides a method including:providing the parameters of compatable virtual image sourcing from firstimaging modality, such as, but not limited to, DDR—to fluoroscopy;determining an object size on virtual image, such as, but not limitedto, ribs width on DDR at specific location; providing the pose and fieldof view of a virtual camera, such as, but not limited to, a virtualfluoroscopic camera, projecting first imaging modality to second imagingmodality such as fluoroscopic camera calculated from calibrationprocess; determining the object size on the virtual image, such as ribswidth on DDR at specific location; calculating the depth (for example,but not limited to, distance of the specific object or object area fromfluoroscopic X-ray source) through comparison between the known objectsizes sourced from first image (e.g. CT image) to the one measured onsecond image (e.g. fluoroscopic image). In some embodiments, the objectsize is determined from technical specification instead of or inaddition to the measurement on compatible virtual image, such as toolrigid part length or width. In some embodiments, the catheter-type toolis designed to allow the calculation of trajectory as a combination ofdepth distances from second imaging modality camera center.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be further explained with reference to theattached figures. The figures constitute a part of this specificationand include illustrative embodiments of the present invention andillustrate various objects and features thereof. Specific functionaldetails disclosed herein are not to be interpreted as limiting, butmerely as a representative basis for teaching one skilled in the art tovariously employ the present invention.

FIG. 1 is a flow chart illustrating an embodiment of the presentinvention, showing a surgical and diagnostic procedure flow chart.

FIG. 2 is an illustration of an embodiment of the method of the presentinvention (e.g., showing an augmented fluoroscopy system and data flow).

FIGS. 3A and 3B are images illustrating an embodiment of the method ofthe present invention.

FIG. 4 is a flow chart showing an embodiment of the method of thepresent invention (e.g., an anatomical structure enhancement flowchart).

FIG. 5 is an illustration showing an embodiment of the method of thepresent invention, illustrating three intensity measurements of themethod of the present invention: (A) shows a pattern obtained from areference imaging modality; (B) shows a signal from an intraoperativemodality; and (C) shows an augmented signal from intraoperativemodality. This illustration shows an embodiment of the method of thepresent invention, where the intensity measurements can be used for fineregistration (i.e., template matching), based on at least one signalenhancement.

FIGS. 6A and 6B is a schematic drawing showing an embodiment of themethod of the present invention, illustrating a fluoroscopic image.

FIG. 7 is an embodiment of the method of the present invention,illustrating a registration step using (1) information pertaining to abronchial airway tree, where the information is extracted from apreoperative image (e.g., a 2-dimensional or a 3-dimensional image;e.g., a CT scan) and (2) information pertaining to at least one airway,where the information is extracted from a fluoroscopic image(s) by useof an augmented bronchogram.

FIG. 8 shows an embodiment of the method of the present invention,illustrating a fluoroscopic image directly after injecting (e.g., 0seconds after injecting) an area with a radiopaque substance.

FIG. 9 shows an embodiment of the method of the present invention,illustrating a fluoroscopic image of an area 30 seconds after beinginjected with a radiopaque substance (e.g., the image appears blurred).

FIGS. 10A, 10B, and 10C show embodiments of the method of the presentinvention, illustrating navigating through at least one bronchus and/ordifferent bronchi, and recording a fluoroscopic image of each navigatingevent.

FIG. 11 shows an embodiment of the method of the present invention,illustrating an augmented bronchogram generated/derived from acombination of images (e.g., but not limited to, FIGS. 10A, 10B, and10C), where the images contain a visible instrument in, e.g., but notlimited to, at least one bronchus.

FIG. 12 shows an embodiment of the method of the present invention,illustrating a straight instrument section projected to a fluoroscopeimage plane.

FIG. 13 shows an embodiment of the method of the present invention,illustrating recovery of depth information related to an anatomical path(e.g., a bronchus/i).

FIG. 14 shows a navigation catheter having an anchor (e.g., disposableor non-disposable catheter) for use in an embodiment of the method ofthe present invention.

FIGS. 15A and 15B are images showing an embodiment of the resultsobtained from using the method of the present invention. FIG. 15A is afirst image (e.g., an original image) and FIG. 15B is a second imagehaving a highlighted section (e.g., shown in a dashed circle).

DESCRIPTION

The present invention will be further explained with reference to theattached drawings, wherein like structures are referred to by likenumerals throughout the several views. The drawings shown are notnecessarily to scale, with emphasis instead generally being placed uponillustrating the principles of the present invention. Further, somefeatures may be exaggerated to show details of particular components.

The figures constitute a part of this specification and includeillustrative embodiments of the present invention and illustrate variousobjects and features thereof. Further, the figures are not necessarilyto scale, some features may be exaggerated to show details of particularcomponents. In addition, any measurements, specifications and the likeshown in the Figures are intended to be illustrative, and notrestrictive. Therefore, specific structural and functional detailsdisclosed herein are not to be interpreted as limiting, but merely as arepresentative basis for teaching one skilled in the art to variouslyemploy the present invention.

Among those benefits and improvements that have been disclosed, otherobjects and advantages of this invention will become apparent from thefollowing description taken in conjunction with the accompanyingfigures. Detailed embodiments of the present invention are disclosedherein; however, it is to be understood that the disclosed embodimentsare merely illustrative of the invention that may be embodied in variousforms. In addition, each of the examples given in connection with thevarious embodiments of the invention which are intended to beillustrative, and not restrictive.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrases “in one embodiment” and “in someembodiments” as used herein do not necessarily refer to the sameembodiment(s), though it may. Furthermore, the phrases “in anotherembodiment” and “in some other embodiments” as used herein do notnecessarily refer to a different embodiment, although it may. Thus, asdescribed below, various embodiments of the invention may be readilycombined, without departing from the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” include pluralreferences. The meaning of“in” includes “in” and “on.”

As used herein, “coarse registration” refers to a rough alignment of apreoperative and an intraoperative image. In some embodiments of themethod of the present invention, coarse registration uses globalinformation and does not take into account local tissue deformationcaused by breathing, instrument movement, pose difference betweenpreoperative and intraoperative images, etc.

As used herein, an “element” refers to a unit of anatomy that has acommon mechanical characteristic, for example, a mechanical property(e.g., but not limited to, a rigidity of movement, flexibility,strength). In some embodiments, elements can be, but are not limited to,bronchi, vessels, ribs, image patterns, etc.

As used herein, “fine registration” refers to the registration of localtissue (e.g., but not limited to, soft tissue) around an area ofinterest of a first image (e.g., a preoperative image), whichcorresponds to an area of a second image (e.g., an intraoperativeimage). In some embodiments of the method of the present invention, fineregistration is a technique/method designed to correct local tissuedeformation and/or relative tissue movement (e.g., but not limited to,movement divergence between ribs and lungs during breathing) inside anarea of interest, e.g., but not limited to, a local proximity of a tooltip, a pre-marked nodule area, etc. In some embodiments, fineregistration further allows for improvement of local registrationaccuracy over coarse registration in an area of interest, while coarseregistration output, such as transformation matrix, projectedprimitives, output images, etc., are supplied as input for use of thefine registration.

As used herein, “mapping” refers to transferring a plurality of elementsfrom a first image of a first imaging modality to a second image of asecond imaging modality. In some embodiments, mapping can include: (1)identifying a plurality of elements of a first image (2) identifying aplurality of elements of a second image, (3) pairing the plurality ofelements of the first/second image to a corresponding plurality ofelements of a second/first image, (4) registering (i.e., registration) aplurality of elements of the first/second image to corresponding pairsof the plurality of elements of a second/first image. In someembodiments, the registering is performed by fine and/or coarseregistration. As a non-limiting example, mapping can include (1)identifying a plurality (e.g., but not limited to, 2, 3, 4, 5, 6, 7, 8,9, 10, etc., elements) of elements (e.g., bronchi, ribs, etc.) from afirst image (e.g., a CT image), (2) identifying a plurality offluoroscopic elements on the first image (e.g., a CT image) and aplurality of fluoroscopic elements on the second image (e.g., afluoroscopic image) (3) pairing a subset of the plurality of elementsthat are corresponding elements (i.e., to bronchi, ribs) on a secondimage, (4) registering the elements to the corresponding pairs of theelements on the second image, where the mapping results in arepresentation of the airway of the first image, or any combinationthereof. In some embodiments, an image can be derived from a raw image,e.g., but not limited to, a DDR image, an edited image, a processedimage, etc.

In some embodiments, although the term “preoperative image” is used todescribe the invention it will be apparent to one skilled in the artthat the same concept can be applied when the reference image such asCT, MRI or X-Ray Radiograph imaging is acquired intraoperatively. Insome embodiments, the method of the present invention is applicable forthe imaging performed with or without contrast medium.

In some embodiments, the present invention is a method that allows usinga first imaging modality (such as CT, MRI, etc.) and planninginformation by generating an augmented image using a second imagingmodality, such as, but not limited to, fluoroscopy, digital subtractionangiography (DSA), etc. In some embodiments, the method further includeshighlighting an area of interest and/or structures. In some embodiments,the method can include additional imaging and/or planning information,where the additional imaging and/or planning information can beoriginated/generated from a first imaging modality, and can includesuperimposing, as non-limiting examples: (i) a first imaging modalityfor use in obtaining at least one first image of chest; (ii) manualand/or automatic planning of a surgical procedure through defininglandmarks, area of interest, incision points, critical structures,bifurcations, anatomical organs, etc.; (iii) at least one second imageobtained from second imaging modality, such as, but not limited to,fluoroscopy and/or DSA, and generation of compatible virtual image, suchas a digitally reconstructed radiograph (DRR), from a first imagingmodality; (iv) a map (“mapping”) of planning data to at least one objectand/or structure on the compatible virtual image; (v) a registration ofat least one second image or video frame from second imaging modality tofirst image or its portion sourced from first imaging modality; (vi)planning data identified from the compatible virtual image, sourced fromfirst imaging modality to at least one second image from second imagingmodality by means of image registration; (vii) planning data mapped fromthe compatible virtual image, sourced from first imaging modality to atleast one second image from second imaging modality by means of imageregistration; (viii) a highlighted area of interest, e.g., but notlimited to, at least one anatomical structure on the at least one secondimage sourced from second imaging modality to obtain at least one thirdimage, wherein the at least one third image is augmented, or anycombination thereof.

In some embodiments, the method further includes superimposing of atleast one image or a derivative of the at least one image, a portion ofthe at least one image or image based planning information sourced fromthe first imaging modality. In other embodiments, the method furtherincludes navigation and guidance instructions that aid movement ofmedical instrument. In some embodiments, the method further includesguidance for positioning the second imaging modality, such as use of afluoroscopic C-Arm, to allow maintaining optimal visibility for an areaof interest. In some embodiments, the method further includes trackingof an anatomic structure(s) on subsequent frames from second imagingmodality, such as, but not limited to, fluoroscopic video, havingsubstantially the same acquisition parameters, where the acquisitionparameters can include, but are not limited to, mode, position, field ofview, to result in generating a augmented fluoroscopic image, where theaugmented fluoroscopic image is generated by suppression of a staticanatomic structure(s) and/or improving signal to noise ratio ofunderlying soft tissue. In some embodiments, the method includesperforming multiphase registration, where at least one static object(s)having small movement(s) (e.g., but not limited to, 2-5 centimeters),such as, e.g. but not limited to ribs, are first registered. In someembodiments, after the static object(s) are first registered, moredynamic objects such as, but not limited to, diaphragm, bronchi, bloodvessels, etc. are registered in the following registration iterations.In some embodiments, the method further includes the interferingstructures (e.g., any structure that could interfere with an anatomicalfocus of a procedure (e.g., but not limited to removing ribs from animage focusing on vessels)) being deemphasized.

In some embodiments, the method of the present invention allows for thegeneration of at least one augmented third image, such as, but notlimited to, an intraoperative fluoroscopic image, a DSA image, etc.,having a highlighted area of interest and/or structures that caninclude, but is not limited to: (i) using at least two intraoperativeimages with known relative movement and/or rotation to allow for thegrouping of pixels of the at least two intraoperative images accordingto the movement variation and/or intensity values of the at least twointraoperative images; (ii) performing registration and/orcross-correlation between at least two sequential intraoperative imagesto reconstruct structures in the area of interest; (iii) differentiatingmoving and static structures in the area of interest based on userdemand; (iv) highlighting anatomical structures an intraoperative image,or any combination thereof.

In some embodiments, the method of the present invention furtherincludes using an x-ray radiographic image of a patient's chest, whilethe x-ray radiographic image can serve as a reference image for enablingan enhancement of at least one anatomical structure on a second image byuse of an analogous process, i.e., cross-correlation of the informationfrom radiographic image obtained with different energy levels.

In some embodiments, the present invention is an augmented fluoroscopydevice that allows for the generation of at least one augmentedfluoroscopy image, where the augmented fluoroscopy device can include,but is not limited to: (i) a video and image processing unit; (ii) avideo input card and/or externally connected device configured to inputvideo signal from a fluoroscopic device; (iii) 3D planning input ininternal and/or DICOM format; (iv) augmented video signal output, or anycombination thereof.

In some embodiments, the device of the present invention is integratedwithin a fluoroscopic device (i.e., as a module) to obtain RAW data as asignal, and includes a RAW data input card. In some embodiments, thedevice has a RAW data card instead of a video input card. In someembodiments, the present invention is integrated within a Cone-beam CTsystem.

In some embodiments, the present invention is a method for highlightinga tissue or an anatomical structure, where the method can include: (i)selecting the volume of interest on the image sourcing from firstimaging modality, such as, but not limited to, CT and/or MRI; (ii)acquiring an image from a second imaging modality; (iii) performingcoarse registration between a second imaging modality and a firstimaging modality to identify the pose of a virtual camera in the secondimaging modality correspondent to the one of second imaging modality;(iv) producing at least one pattern from first the imaging modality forthe anatomical structure around a volume of interest is produced; (v)identifying a matching pattern in the second imaging modality using asingle pattern or multiple patterns produced from the first imagingmodality; (vi) highlighting (i.e., enhancing) a matching pattern fromthe second imaging modality to enhance the anatomy in the volume ofinterest on third imaging modality, or any combination thereof.

In some embodiments, the method includes finding and suppressinganatomic structures located outside the area of interest.

In some embodiments, the present invention includes a method of objectdepth calculation that includes, but is not limited to: (i) providingparameters of compatible virtual image sourcing from the first imagingmodality, (as a non-limiting example, the first imaging modality can be,but is not limited to, DDR—to fluoroscopy); (ii) determining the objectsize on a virtual image, such as ribs width on DDR at a specificlocation; (iii) providing the pose and field of view of the second image(as a non-limiting example: a fluoroscopic camera calculated from acalibration process); (iv) calculating the depth (such as, but notlimited to, a distance of a specific object or an object area from afluoroscopic X-ray source) by use of a comparison between (a) the knownobject sizes sourced from first image (e.g., but not limited to, a CTimage) to (b) an object measured on a second image (e.g., but notlimited to, fluoroscopic image), or any combination thereof.

In some embodiments, the object size is determined from: (1) a technicalspecification and/or (2) the measurement on a compatible virtual image,such as, but not limited to, a rigid tool part length and/or width. Insome embodiments, the method includes a tool that is designed to allowthe calculation of a trajectory as a combination of depth distances froma second imaging modality camera center.

In some embodiments, the invention provides a device and a method thatextend visualization capabilities of fluoroscopic imaging modality thatis widely used in diagnostic and treatment medical procedures. In someembodiments, the proposed method, called herein “augmented fluoroscopy,”allows enhancing visualization of a specific region of interest withinthe internal structures of the patient being evaluated in real time. Insome embodiments, the method of the present invention is utilized forsoft tissue visualization. In some embodiments, the method allows for apractitioner (e.g., but not limited to, a doctor, a nurse, a specialist,etc.) to have an increased control over the fluoroscopic visualizationcapabilities in medical procedures (e.g., for use in soft tissuevisualization). In some embodiments, use of the method of the presentinvention by trainees reduces the learning curve (e.g., but not limitedto, decreases training time, decreases miscalculations, etc.).

In some embodiments, the device presented in this invention includes thefollowing functions: signal input, processing, and display capabilities,where the functions can be installed in, e.g., a procedure room. In someembodiments, the invented device is configured to integrate signals fromexisting imaging equipment to provide an advanced visualizationcapability(ies). In some embodiments, the present invention is astand-alone device. In some embodiments, the present invention is atleast one module and is integrated inside the current equipment.

In some embodiments, the method of the present invention includesperforming a preoperative planning using preoperative imaging modalitysuch as, but not limited to, a CT scan or a MRI. In some embodiments,the performed preoperative planning can be used to define the area ofinterest and/or mechanical properties of the tissue that can be enhancedduring real-time fluoroscopy. In some embodiments, the method of thepresent invention, in addition to enhancement/highlighting of the areaof interest on an intraoperative fluoroscopic image, can generate anoverlay on an intraoperative fluoroscopic image. In some embodiments,the overlay can include: the location information of internal andexternal landmarks together with anatomic structures such as lesionand/or resection boundaries, incision points, bronchial airways, bloodvessels, etc. In some embodiments, the method includes: (i) performingpreoperative planning and (ii) using the preoperative plan during adiagnostic procedure and/or a treatment procedure. In some embodiments,use of the method of the present invention improves the efficacy andsafety of diagnostic and/or treatment procedures.

In some embodiments, the present inventions disclosed herein relate tothe aspects of augmented fluoroscopy device and method that allowshighlighting the elements or area of interest of the fluoroscopic imagesin real time. Exemplary embodiments of highlighting include optionalsuperposition (e.g., but not limited to, preoperative planning elementsover static or dynamic fluoroscopic images used for diagnostic and/ortreatment procedures). In some embodiments of the method of the presentinvention, highlighting methods include: (i) bolding a selected area,(ii) coloring a selected area (e.g., selecting an area and placing apigment (e.g., but not limited to, yellow, blue, red, green, etc.) on agrayscale image, (iii) enhancing an image of a tissue/area (e.g., seeFIG. 3, where an “augmented image” is an “enhanced image”), (iv)super-positioning a graphic over a fluoroscopic image (e.g., but notlimited to, super-positioning a boundary (e.g., a dotted line, a dashedline, etc.) over a selected area of a CT scan), or any combinationthereof. In some embodiments, highlighting can be performedautomatically, semi-automatically, manually, or any combination thereof.

Conventional fluoroscopy is typically used to obtain real-time movingimages of the internal structures of a patient during medicalprocedures. Conventional fluoroscopy is a visualization and validationimaging tool for guiding medical instruments inside a body (e.g., butnot limited to, a human body). Although the bone tissue and medicalinstruments such as, but not limited to, catheters, biopsy tools,surgical instrument, calibration tool, etc., are clearly visible on afluoroscopic image, the features of lower density matter such as softtissue, blood vessels, suspicious nodules etc., are difficult toidentify with conventional fluoroscopy. Taking lung cancer diagnosticprocedures as an example, a CT scan is usually acquired, prior toprocedure. While the pulmonary nodule is clearly observed on the CT scanit cannot be clearly specified on the fluoroscopic image in most ofthese cases. Prior to a diagnostic and/or a treatment procedure, ahealth care professional (e.g., a physician) typically studies apreoperative CT scan and/or a MRI image to identify the area of interestthat needs to be addressed during an incoming procedure. Using thethree-dimensional (“3D”) imaging information and professionalknowledge/experience, a physician plans the incoming procedure withoutan actual detailed documentation of such a plan.

During the actual diagnostic or treatment procedure physician isfrequently using a fluoroscope to verify/identify the position and/oroperation of the diagnostic and surgical instrument. Since the targetarea is not clearly specified on the fluoroscopic image, the physiciancan be required to guess/estimate the location of the target area.Moreover, since the fluoroscopic image represents accumulatedinformation from the x-rays passing through the patient, as the x-raysare attenuated by varying amounts when interacting with the differentinternal structures of the body, the low-density soft tissues areoccluded by high-density tissue. In addition, the three-dimensionalinformation is missing from a fluoroscopic image. As a result, there ishigh probability of user errors caused by misinterpretation of visualinformation displayed on fluoroscopic images. Finally, the typicalapproach generally results in a the low diagnostic yield (i.e., thelikelihood that a diagnostic procedure will provide the informationneeded to establish a definitive diagnosis) of 35%, substantially largerresection area margins (e.g., but not limited to, 10%, 20%, 30%, 40%,50% larger), substantially longer procedure time and inconsistentresults within the same medical facility while targeting soft tissuearea or nodules through the conventional fluoroscopy.

An electromagnetic navigation system (ENB) may be used in the method ofthe present invention to support inter-body navigation. The ENBtypically uses preoperative static CT images.

The method of the present invention uses real time fluoroscopic images(i.e., not static images). In some embodiments, the present invention isa device configured to achieve a real time modality that allows auser/practitioner to visualize (effectively) the soft tissue target areaof diagnostic and/or treatment procedure with a diagnostic or surgicalinstrument. In some embodiments, real-time visualization isadvantageous, since preoperative static image information, such as CT orMRI, is inaccurate for localization of instruments relatively to thetarget area due to significant movement and/or deformation of the lungtissue during breathing, where deformation is caused by an advancementof a diagnostic instrument or a surgical instrument inside a patient(e.g., a human body) in addition to potentially substantially dissimilarpatient conditions compared between (a) a preoperative CT imaging and(b) actual diagnostic or treatment procedure.

In some embodiments, the method of the present invention can include useof a third imaging modality configured to use a second imaging modality(e.g., but not limited to, real time fluoroscopy) during a diagnostictreatment or a treatment procedure in conjunction with use of a firstimaging modality (e.g., but not limited to, preoperative CT). In someembodiments, the method can include a third imaging modality configuredto produce a third image having highlighted elements/features ofinterest (i.e., augmented image) during a diagnostic and/or a surgicalprocedure. In some embodiments, the method can facilitate a reduction inoperation time and/or an improvement in the learning curve of suchprocedures (e.g., for a nascent practitioner).

In some embodiments, the method of the present invention can be usedduring a surgical procedure and/or guiding under real-time visualizationof an area of interest.

In some embodiments, the method allows a practitioner to controlvisibility of specific elements of an area of interest on a third image(e.g. fluoroscopic image) by adding at least one three-dimensionalaspect of information to a second image (e.g. conventional fluoroscopicimage). In some embodiments, the method can aid a user to focus on anarea of interest (i.e., the correct area of interest required during asurgical procedure), including, for example, an inspection of adjunctivestructure around the area of interest, such as, but not limited to,blood vessels, bronchial airways, etc. In some embodiments, the methodof the present invention includes suggesting to a user an optimalfluoroscopic angle to increase visibility of a lesion at the time of adiagnostic and/or treatment procedure, where the suggestion is based onat least one DDR preoperative image.

In some embodiments, the method of the present invention allows forproviding increased control to a physician during a surgical procedure,where the control includes sufficiently improving the physician'sability to accurately identify a treatment area and/or at least onecritical structure(s) relatively to the diagnostic instrument and/orsurgical instrument according to pre-operative planning andthree-dimensional imaging data.

In some embodiments, the method of the present invention uses a hardwaredevice having integrated software algorithms that are configured toallow for an integration and processing of first images (e.g.pre-procedure) and second images (e.g. intraoperative fluoroscopic), andrendering real-time or offline images of a third image (e.g. augmentedfluoroscopy) on an output (i.e., a result).

In some embodiments, the method of the present invention uses an angularmeasurement device/sensor (e.g., a right angle sensor, an accelerometer,gyroscope, etc.) that is configured to allow for determining a spatialrelative angle and/or position (pose) between: (a) the C-Arm offluoroscope and (b) the patient.

In some embodiments, the method of the present invention can utilize asteerable catheter configured to allow measuring a depth inside apatient (e.g., but not limited to, within a patient's chest) and/or adistance from a fluoroscopic camera.

In some embodiments, the device and method of the present inventionprovide a real-time third imaging modality (e.g. augmented fluoroscopicmodality) to allow for use of (a) information originated from a firstimage (e.g. pre-operative CT image) and (b) information (e.g.,decisions) made during the planning phase for highlighting an area ofinterest (i.e., providing an augmented image), optionally including adisplay of (a) the information originated from the first image and/or(b) information generated during the planning phase over second image(e.g. fluoroscopic image).

In some embodiments, the methods of the present invention can be used toassist the diagnostic and/or treatment procedures involving soft movingtissues such as, but not limited to, lung, liver, kidney, etc. In anexemplary embodiment, in pulmonology, peripheral nodules can behighlighted on a fluoroscopic image and/or a digitally reconstructedradiograph (DRR) image of the peripheral nodules can be superimposedover the fluoroscopic image in real time. In some embodiments, theapproach of using three-dimensional CT image to highlight the area ofinterest on the two-dimensional (“2D”) fluoroscopic image is applicableto other medical applications.

In some embodiments, the method of the present invention can be usedwith a Cone Beam CT device. In some embodiments, combining the method ofthe present invention with a Cone Beam CT device allows for greaternavigation accuracy, automatic fluoroscopic pose control, radiation dosereduction, etc.

In some embodiments, the method of the present invention allows apractitioner to navigate and/or operate a medical instrument(s)according to real time information highlighted on third image (e.g.fluoroscopic image/augmented image), where the third image can includesuperimposed anatomical and/or planning data extracted from apre-operational image.

In some embodiments, the method of the present invention provides areal-time third image (e.g. fluoroscopic image/augmented image) of anactual surgical instrument and highlighted area of interest and/oranatomical elements. In some embodiments, the method can provide anoverlaid targeted anatomical feature(s) on the augmented image. In someembodiments, the method can provide planning information, such as, butnot limited to, incision points, cutting area boundaries, referencepoints, etc., on the augmented image.

In some embodiments, the method and device of the present inventionallow a user/practitioner to combine multimodal imaging information andutilize previously acquired three-dimensional volume data to highlightmoving and static soft tissue area (i.e., generate an augmented image).

In some embodiments, the method of the present invention includesproducing an augmented fluoroscopy image that provides to auser/practitioner an identifying structure(s) on the augmentedfluoroscopic image, which is generated by a movement variabilityanalysis of groups of pixels (e.g., different groups of pixels) on afluoroscopic video and/or sequential fluoroscopic image(s). In anexemplary embodiment, the soft tissue lesion inside the lungs moves in adifferent direction in comparison with the ribs, and the amplitude ofsoft tissue movement is typically greater than one of the ribs,resulting in a projected movement of the soft tissue and rib structureshaving a difference as measured by the fluoroscopic video frames. Insome embodiments, the measured difference combined with the informationof each pixel attenuation value allows for the grouping of pixels intophysical structures and/or objects. In some embodiments, when groupedinto objects, the physical structures can be highlighted or deemphasizedon the fluoroscopic image in reference to a medical applicationdetermined by a user/practitioner. In some embodiments, the augmentedfluoroscopic image can be further enhanced by extracting the objectinformation from the sequence of fluoroscopic images, which can beoptionally refined with the information provided by a preoperative imagesuch as, but not limited to, CT, MRI, chest x-ray radiographic image, orany combination thereof.

In some embodiments, the method of the present invention includes anautomatic calibration of at least one static fluoroscopic image and/orvideo frame from a real time video. In another embodiment, the methodincludes (i) generating a prediction of the quality of specificanatomical structure or visibility of an area of interest duringintraoperative fluoroscopy at various angles and (ii) recommendingangles to use a fluoroscopic C-Arm for improving visibility of thespecific anatomical structure or area of interest, which providesguidance to a user and achieves increased visibility of thestructure/area of interest, e.g., relative to the background of animage.

In some embodiments, the method of the present invention providesprocessing the RAW data obtained from a fluoroscopic device by changingan existing automatic gain algorithm integrated with the fluoroscopicdevice, based on the whole fluoroscopic image. In some embodiments, themethod includes the use of a region-based gain calculation algorithm. Insome embodiments, a specified region-based gain calculation algorithm isderived from the knowledge of correspondent three-dimensional anatomy,where the correspondent three-dimensional anatomy is obtained from CT orMRI images, around the area of interest and includes evaluating thephysical properties of the area of interest. In some embodiments, themethod provides for a specific signal processing, which reduces a lossof information provided on the resulting fluoroscopic image in thetarget area (i.e., augmented image), and can also resulting in anincrease of visibility of the target area.

In some embodiments, the method and device of the present invention canbe used to maintain/generate an accurate registration (i.e., coarseregistration and/or fine registration) between two or more operativereal-time video images and/or static preoperative images.

In some embodiments, the method and device of the present invention caninclude the use of pre-operative data (i.e., decisions/informationgenerated by a user/practitioner), where information is displayed on thescreen, and the resolution and/or quality of the displayed informationcan be dynamically determined on an application-specific oruser-specific basis.

In some embodiments, the present invention is a method that uses ahardware device having integrated software algorithms configured toprovide an input from a first imaging modality (e.g. pre-procedureimage) and second imaging modality (e.g. intraoperative fluoroscopicimage) that generates third imaging modality images (e.g. augmentedfluoroscopic image) as output.

In some embodiments, the method of the present invention provides areal-time output calibrated image with configurable display elements andoutput video format.

In some embodiments, the method of the present invention can use ahardware device with integrated software algorithms that has standaloneand/or modular architecture.

In some embodiments, the method of the present invention uses a hardwaredevice that is configured to provide an angular measurement determiningrelative spatial pose between the fluoroscope C-Arm and patient body toa user. In some embodiments, the device is applicable for thosefluoroscope models where the angular information is unavailable orinaccessible during procedure.

In another embodiment, the method of the present invention can includereconstructing at least one anatomical structure in a three-dimensionalspace from several fluoroscopic images (e.g., 2 images, 3 images, 4images, 5 images, 6 images, 7 images, 8 images, 9 images, 10 images,etc.) by using the correspondent three-dimensional anatomical structuresderived from preoperative images, e.g., CT scans).

Referencing FIG. 1 there is shown a flowchart that illustrates method100 of an embodiment of the present invention.

At 101 of the method 100 of an embodiment of the present invention,first image (e.g. preoperative image, such as CT or MRI), is acquiredand transformed into 3D space, which is used during surgical treatmentor diagnostic procedure to plan the treatment and/or diagnosis.

At 102 of the method 100 of an embodiment of the present invention, thepractitioner (for example, but not limited to, pulmonologist or surgeon)performs pre-procedure planning on the pre-procedure data acquired at101, during which the practitioner marks the area of interest (e.g., theboundaries of the area to biopsy or resect around the suspicious lesion,the approach or incision points for preferred tool introduction,critical structures (e.g., but not limited to, major blood vessels,restricted area)), the preferred pathway to approach the area ofinterest. In some embodiments, the procedure (i.e., 102) may beperformed manually and/or semi-automatically, such as when part ofinformation is automatically identified by computer software.

In some embodiments of the present invention, once the planning iscompleted, at 104 the information is processed to map (i.e., “mapping”)and/or identify (i.e., “identifying”) the area of interest, wheremapping and/or identifying allows for planning elements in a 3D spaceand/or identify major anatomical structures. In some embodiments,information gathered from mapping (i.e., “mapping information”) istransferred from (a) image sourcing from a first imaging modality to (b)an image sourcing from a second imaging modality. In some embodiments,the mapping information is transferred after the coarse and/or fineregistrations are performed on the first image source and the secondimage source. In some embodiments, an image source (e.g., but notlimited to, a first image source) can be use/reused for highlightingpurposes during second imaging modality operation (e.g., but not limitedto, intraoperative fluoroscopy).

Non-limiting examples of mapping or identifying techniques for bodyorgans are disclosed in “Automatic localization of solid organs on 3D CTimages by a collaborative majority voting decision based on ensemblelearning” by Zhou X, Fujita H, Comput Med Imaging Graph, 2012, which isherein incorporated by reference in its entirety. For example, alocation of a target organ in a 3D CT scan can be presented as a 3Drectangle that bounds the organ region tightly and accurately (e.g.,serving as a boundary for at least one organ). For example, the locationof a target organ-specific 3D rectangle (e.g., but not limited, to abound rectangle) is detected automatically. Multiple 2D detectors aretrained using ensemble learning and the outputs of the multiple 2Ddetectors are combined using a collaborative majority voting in 3D tolocalize an organ(s). For example, the location detection of differentinner organs can be used separately and/or independently. The exemplarymethod includes treating 3D organ localization in a 3D CT scan asdetecting several independent 2D objects in a series of 2D image slices,where the method can (i) reduce the feature dimension (3D to 2D) and(ii) increase the number of training samples (e.g., one 3D trainingsample consists of a large number of 2D training samples) duringensemble learning. The exemplary method can increase the robustness ofthe trained detector for unknown samples according to Occam's razor. Forexample, for an unknown 3D CT scan, the exemplary method appliesdifferent 2D detectors to each voxel independently to detect a number of2D candidates of a target along three orthogonal directions and votesthose 2D candidates back to the 3D space. The existence and approximatecenter position of the target can be determined by checking the mutualconsent of the responses all 2D detectors and selecting the majority ofthe range of the related 2D candidates in the 3D voting space as thetarget location.

Non-limiting examples of mapping or identifying techniques for bodyorgans are also disclosed in “Registration of a CT-like atlas tofluoroscopic X-ray images using intensity correspondences,” M. Sc thesisby Aviv Hurvitz, supervised by Prof. Leo Joskowicz, The Rachel and SelimBenin (School of Computer Science and Engineering The Hebrew Universityof Jerusalem, Israel, August, 2008), which is herein incorporated byreference in its entirety. This exemplary method allows forintraoperative localization of bones, where the method does not requireany preoperative images, and is less invasive than many alternatives.For example, in the preoperative stage, a CT-like intensity atlas of theanatomy of interest is constructed from sample CT images. In theintraoperative stage, a novel 2D/3D deformable registration algorithm isused to register the atlas to Fluoroscopic X-ray images of the patientanatomy. The registration algorithm is configured to establishintensity-based correspondences between the atlas's template bonesurface and bone contours in the fluoroscopic X-ray images. Theregistration algorithm further is configured to search for the boneshape and pose that minimize/reduce the distances between pairedfeatures. The algorithm iteratively is configured to refine the boneshape and pose estimates until the bone shape and the pose estimate(s)converge.

In some embodiments, the method includes generating an augmented 3Dfluoroscopic image by use of a 2D fluoroscopic image by matching eachpixel on the 2D fluoroscopic image to 3D structures sourced from a CTscan. The method of the present invention does not utilize tracingelements and/or markers, such as, but not limited, to radiopaque markertethered to a device, a radiopaque particulate spray, an inflatableradiopaque balloon, a radiopaque filament, during a registration.

In embodiments, the method of the present invention can generate: (i)visualization data that shall be displayed during surgical procedure;(ii) a recommended pathway for introduction of at least one medicalinstrument; (iii) guidance instructions based on anatomic knowledge andprocedure details; (iv) recommended angles or pose for C-Arm, as toresult in optimizing the area of interest visibility, or any combinationthereof.

In some embodiments, the fluoroscopic image is acquired at 106 duringprocedure while medical instrument is introduced into the area ofinterest. In some embodiments, the fluoroscopic image can be acquired assingle image and/or video.

In an embodiment, the generated fluoroscopic image and/or video isintroduced into the processing unit 218, FIG. 2 as an input forfluoroscopic image processing 108. In the embodiment, the pose betweenthe Fluoroscopic C-Arm 209, FIG. 2 and patient 214, FIG. 2 is eithertransmitted from outside or calculated by processing unit. In theembodiment, the compatible digital reconstructed radiograph (DRR) imageis generated from a pre-procedure image using substantially the samepose of a virtual C-Arm and substantially the same camera parameters asthe actual Fluoroscope. In some embodiments, the image is calibrated,where “calibrated” means being adjusted for fluoroscopic imagedistortion and compensated for x-ray energy difference between thefluoroscope and CT at the intensity values according to the prior artknowledge of X-ray radiometry.

In some embodiments, the following references discuss DDR simulation,calibration and registration to actual fluoroscopic images: “2D/3D ImageRegistration on the GPU.” Alexander Kubias, University ofKoblenz-Landau, Koblenz, Germany, Thomas Brunner, Siemens MedicalSolutions, Forchheim, Germany, 2007, which is hereby incorporated byreference in its entirety. For example, this exemplary method performsthe rigid 2D/3D image registration efficiently on the GPU [graphicsprocessing unit]. Both parts of the registration algorithm, i.e. the DRRgeneration and the computation of the similarity measure, are executedon the GPU. Additionally, “2D/3D Registration for X-ray GuidedBronchoscopy using Distance Map Classification,” by Di Xu, Sheng Xu,Daniel A. Herzka, Rex C. Yung, Martin Bergtholdt, Luis F. Gutiérrez,Elliot R. McVeigh, is hereby incorporated by reference in its entirety.For example, the registration algorithms can be grouped into twocategories: (1) intensity based and (2) feature based, where thefeature-based registration can be used in connection with the method ofthe present invention. For example, the edges of the ribs and spine canbe extracted from the X-ray and/or CT images. A distance map can furtherbe generated for a plurality of (e.g., but not limited to, each recordededge point, which can result in using all edge points) the edge pointsof the X-ray image to facilitate/allow the 2D/3D registration byattracting the edge projections of the CT image to the closest edges inthe X-ray image. When the distance map does not have any orientationinformation of the edges, mis-registration can occur between the edgesof different structures. Mis-registration can be reduced by usingorientation dependent distance maps to achieve more robust registrationwith improved capture range and accuracy.

In some embodiments, the map generated in 104 is used to provide spatialinformation for each projected element on the DRR image. In someembodiments, the registration is performed between DRR and actualfluoroscopic images. Examples of registration, e.g., feature-based orintensity-based registration, are described in “Automatic registrationof portal images and volumetric CT for patient positioning in radiationtherapy”, (See, e.g., Ali Khamene, Frank Sauer, Medical Image Analysis10 (2006) 96-112), which is hereby incorporated by reference in itsentirety. For example, the feature based registration approach caninvolve a step of feature correspondence between features of each of theimaging modalities participating in registration process. As a result ofthe registration the spatial information generated for DRR image can betransferred onto the actual fluoroscopic image. The 3D spatialinformation added to the actual fluoroscopic image allows implementingcomputer vision approach to the actual fluoroscopic image, thusoperating with objects in 3D space rather than working with 2D image ofpixels. Using this approach allows for each pixel of a fluoroscopicimage to be described by integration of X-ray beam passing through knownanatomic structures.

In some embodiments, the information that was lost during fluoroscopicimage acquisition is restored using the method of the present invention.In some embodiments, the area of interest can be highlighted on theactual fluoroscopic image, while the interfering structures such asbones, heart, blood vessels can be deemphasized. In some embodiments, anadditional improvement of the augmented image quality can be achievedthrough the tracking of sequential video frames, where the movementcharacteristics may vary for different anatomic structures.

The augmented fluoroscopic image or video frame sequence is produced in110 using an embodiment of the method of the present invention. In someembodiments, various elements generated on the planning phase can bedisplayed on augmented fluoroscopic image according to user demand ordepending on system configuration.

FIG. 2 shows a diagram illustrating an embodiment of the presentinvention, showing an augmented fluoroscopy system/method and data flow.

In an embodiment of the present invention for producing an augmentedfluoroscopic image, the method included use of:

-   -   1) C-Arm 202 that is responsible for movement of frame 209 with        attached fluoroscopic pair of X-Ray tube 204 and intensifier        208;    -   2) X-Ray tube 204 that generates X-rays, passing through the        collimator 206, that is designed to narrow the X-ray beams;    -   3) the generated X-ray beam is passing through the patient body        214 attached to the bed 212;    -   4) the attenuated X-Ray beam is further absorbed by X-ray image        intensifier 208 forming the RAW data fluoroscopic image. The        X-ray is converted into the visible image by 208; and/or    -   5) the video signal is constantly captured by camera 210 and        transferred to the monitor 216.    -   6) a planning station 222 that is getting CT image 220 as an        input allows user to plan diagnostic and treatment procedure as        specified by 102, 104 FIG. 1 above;    -   7) a generated planning data, 3D volume data are transferred        into unit 218, where a video signal from 216 or alternatively        RAW data from 208 is constantly transferred to the processing        unit 218;    -   8) the augmented video image is produced by 218 as specified by        108, 110 FIG. 1 and displayed by the monitor 224;    -   9) or any combination thereof.

In an embodiment of the present invention, the following elements wereadded to provide the C-Arm pose measurement: (1) a sensor 211 attachedto frame 209 of C-Arm and/or (2) a reference sensor 213 attached to thepatient body 214 and/or to patient bed 212.

Examples of sensing technologies available for use in embodiments of thepresent invention to allow for evaluation of pose estimation caninclude: an optical sensor, an accelerometer, an electro-magneticsensor, an ultra-sonic sensor, a gyroscopic sensor (e.g., available onthe modern smart phones), etc. An example of use of a pose estimationapproach, which can be used in the method of the present invention, isdescribed in “Robust Multi Sensor Pose Estimation for MedicalApplications” by Andreas Tobergte, Gerd Hirzinger, Intelligent Robotsand Systems, 2009, IROS 2009, IEEE/RSJ International Conference, whichis hereby incorporated by reference in its entirety.

In some embodiments, the method can use a set(s) of markers withpredefined geometric configuration can be attached to the patient bed asdiscussed in “Fast Marker Based C-Arm Pose Estimation” by BernhardKainz, Markus Grabner, and Matthias Rüther, Institute for ComputerGraphics and Vision, Graz University of Technology, Austria, which ishereby incorporated by reference in its entirety.

FIG. 3 shows an exemplary embodiment of the present invention, showingan illustration of an augmented fluoroscopic image. In an embodiment,the diagnostic instrument and bones are clearly seen on the originalimage while the target area is invisible or unclear. In an embodiment,the target area is highlighted on the augmented fluoroscopic image,e.g., on the right. In an embodiment, the method includes highlightingblood vessels, while deemphasizing the bones.

FIG. 4 shows an embodiment of the method of the present invention,showing a flowchart of the method 400. At 401 of the method 400 shows anarea of interest being selected by user on preoperative image, such asCT or MRI prior to diagnostic or treatment procedure. At 403 of themethod 400, the volume of interest is generated on preoperative image.In an embodiment, the volume is generated in such way that theanatomical structures in the area of interest, such as lesion, andadjunctive anatomical structures such as bronchi or blood vessels, willbe detectable on operative image, such as fluoroscopic image. In anexemplary embodiment, for instance, DDR image can be used to evaluatedetectability on fluoroscopic image.

In some embodiments of the method of the present invention, at 405 ofmethod 400, intraoperative image or videos are acquired. In anembodiment, the pose of the intraoperative modality is calculated orrecorded with at least one intraoperative image. In an embodiment, at407 of the method 400, the coarse registration between intraoperativeand preoperative images is performed, e.g., but not limited to,fluoroscopy to DDR, to evaluate a viewpoint of DDR inside a preoperativeimage data, such as, but not limited to, CT volume. An example of coarseregistration is shown in “2D/3D Image Registration on the GPU,” byAlexander Kubias, University of Koblenz-Landau, Koblenz, Germany, ThomasBrunner, Siemens Medical Solutions, Forchheim, Germany, 2007, which ishereby incorporated by reference in its entirety. Some embodiments ofthe method of the present invention use, for example, a rib-based rigidimage registration: For example, using 2D/3D image registration, apreoperative volume (e.g. CT or MRT) is registered with anintraoperative X-ray image. Rigid image registration can be used by themethod of the present invention, where a volume can only be translatedand rotated according to three coordinate axes, where a transformationis given by the parameter vector x=(t_(x), t_(y), t_(z), r_(x), r_(y),r_(z)). The parameters t_(x), t_(y), t_(z) represent the translation inmillimeters (mm) along the X-, Y- and Z-axis, whereas the parametersr_(x), r_(y), r_(z) belong to the vector r=(r_(x), r_(y), r_(z)). Insome embodiments, coarse registration can be performed automatically.

In some embodiments, the method of the present invention can use theregistration techniques disclosed in, “Automatic registration of portalimages and volumetric CT for patient positioning in radiation therapy,”by Ali Khamene, Frank Sauer, Medical Image Analysis 10 (2006) 96-112,which is hereby incorporated by reference in its entirety. In exemplaryembodiments, such registration can be implemented, as a non-limitingexample, as intensity-based and/or as feature based, depending on thespecific medical application. Examples of intensity-based and featurebased registration are described by “Intensity-based Registration versusFeature-based Registration for Neurointerventions” by Robert A., DavidJ. Hawkesb, Medical Vision Laboratory, Dept of Engineering Science,University of Oxford, England, which is hereby incorporated by referencein its entirety.

In some embodiments of the method of the present invention, point-basedregistration can be implemented using known anatomical landmarks on apatient's chest. In some embodiments, at least one known landmark(s) canbe marked on a CT image and/or fluoroscopic image. In some embodiments,special markers can be attached to the patient's chest during procedureto improve/increase detectability on a fluoroscopic image.

In some embodiments, at 409 of the method 400, the set of features orpatterns, depending on desired registration method, is generated from avolume of interest of the preoperative image. In some embodiments, whenthe soft tissue structures of a patient are observed and move relativeto the ribs of the patient, the viewpoint calculated during coarseregistration at 407 is approximated within the known tolerance. In someembodiments, the set of patterns generated at 409 will allow performingthe fine-tuning (i.e., fine registration) of the viewed area in thefollowing step.

In some embodiments, at 411 of the method 400, fine registration isimplemented to find the best fit between each of the features orpatterns, depending on the registration method, generated at 409 andarea of interest on intraoperative image.

In an exemplary embodiment, a fine registration method is illustratedthrough intensity-based fine registration (i.e., template matching),e.g., as shown in FIG. 5, where the approach is initiated with anintensity-based pattern, as shown in FIG. 5A, from a pre-operative or areference imaging modality. In an embodiment, the signal from anintraoperative image, as shown in FIG. 5B, contains noise and scalecorresponding to the pattern shown in FIG. 5A, and is measured withinthe area of interest. In an embodiment, the pattern shown in FIG. 5A ismatched to the pattern from signal FIG. 5B.

An example of a fine registration (i.e., template matching) techniquethat can be used by the method of the present invention is described in:“An Overview of Template Matching Technique in Image Processing” by T.Mahalakshmi, R. Muthaiah and P. Swaminathan School of Computing, SASTRAUniversity, Thanjavur, Tamil Nadu, India, Research Journal of AppliedSciences, Engineering and Technology 4(24): 5469-5473, 2012., which ishereby incorporated by reference in its entirety. Some embodiments ofthe method of the present invention use an area-based approach, whichare also referred to as correlation-like methods or fine registration(i.e., template matching), see, e.g., Fonseca and Manjunath,“Registration techniques for multisensor remotely sensed imagery” PE &RS-Photogrammetric Engineering & Remote Sensing 62 (9), 1049-1056(1996), which describes the combination of feature detection and featurematching. For example, this method is suited for the templates whichhave no strong features corresponding to an image, since the templatesoperate directly on the bulk of values. Matches are estimated based onthe intensity values of both image and template. Techniques that can beused by the method of the present invention include: squared differencesin fixed intensities, correction-based methods, optimization methods,mutual information, or any combination thereof. In some embodiments, themethod of the present invention can perform a fine registrationautomatically.

In some embodiments, the method of the present invention can perform acoarse registration automatically.

In an exemplary embodiment, the method of the present invention canutilize a fine registration method, where the fine registration methodincludes aligning a 2D projection of an anatomical structure from a CTscan obtained through coarse registration with correspondent anatomicalstructure extracted from fluoroscopic image.

At 413 of the method 400 of an embodiment of the present invention, thesignal matching pattern is shown in FIG. 5A. Inside the signal (FIG. 5B)is enhanced to highlight the anatomy found in the area of interest asdrawn by 401. In some embodiments, in addition to highlighting thesignal from intraoperative image, the signal sourcing from referenceimage can be overlaid on the display/image. In another embodiment, thecombination of original signal from intraoperative image, simulatedsignal from reference image and planning information can be displayedaccording to application configuration or upon the user request. In someembodiments, the method shown in FIG. 5C can be alternatively used forsignal suppression.

FIG. 5 shows an illustrative example of fine registration (as shown instep 411 of FIG. 4) (i.e., template matching) of the method of thepresent invention. Although this illustration is shown in one dimensionfor simplicity purposes, the original signals of the embodiment aretwo-dimensional. In some embodiments, steps 411 and 413 of FIG. 4provide the methods using a template-matching registration approach.

The exemplary embodiment shown in FIG. 6, is a schematic drawing of afluoroscopic image, where A, FIG. 6 and B, FIG. 6 represent fluoroscopicimages for two different lung positions during breathing. In theembodiment, the ribs 602 remain almost static while the soft tissuelesions 606 and 608 move substantially between the two breathingpositions. In an embodiment, the tip of the forceps 604 is located inthe close proximity of lesion 606, which results in the forceps movingwith the lesion 606, while the bronchoscope 612, which is located farfrom the lesion, is substantially static and does not substantially movebetween two breathing positions A and B. In an embodiment, the ribintersection area 610 is darker then the rib 502 and can be potentiallyconfused with lesion on the conventional fluoroscopic images. In someembodiments, the analysis of sequential fluoroscopic images A and Ballows to separate substantially static and moving objects, group thestatic and moving objects by (i) movement, (ii) connectivity, (iii)density, or any combination thereof, and/or perform reconstruction ofanatomic structures from a plurality of fluoroscopic images.

In some embodiments, the inventive method can be used for the followingpulmonology-based procedures including, but are not limited to:

-   -   1) Endobronchial diagnostic biopsy, when the pulmonologist first        identifies the lesion under augmented imaging. Then, the biopsy        forceps are advanced to the target site under augmented imaging        to insure the biopsy is taken appropriately;    -   2) Augmented imaging guided percutaneous diagnostic biopsy;    -   3) Wedge resection with VATS or thoracotomy when thoracic        surgeon places markers augmented fluoroscopy guidance prior to        surgical procedure;    -   4) Trans-bronchial needle biopsy direct vision is used to        visualize the lesion and to guide the bronchoscope. The area to        be biopsied is first identified under augmented imaging and then        the scope is advanced as far as possible to the targeted        segment. Using augmented imaging helps to guide the forceps        distally to the target area, beyond the range of direct vision;    -   5) Augmented imaging guided endobronchial or percutaneous        ablation;    -   6) Or any combination thereof.

In some embodiments, the present invention is used to generatemultidimensional images from 2D fluoroscopic images. In someembodiments, a 2D fluoroscopic image is displayed in gray levels andcomprised of pixels. In some embodiments, each pixel represents anintegrated density of at least one tissue while an x-ray generated by anx-ray tube is absorbed by an image intensifier.

In some embodiments, the objects of higher density (e.g., bones andblood vessels) have greater weight on the integrated pixel density(color) in comparison with integrated pixel density of, e.g., air and/orsoft tissue. In some embodiments, automatic gain algorithms implementedfor fluoroscopic devices make at least one high-density tissue visiblewhile reducing the visibility of at least one soft tissue. In someembodiments, at least one suspicious lesion area, although having smallvolume relative to, e.g., bones, has higher tissue density than at leastone normal tissue. In some embodiments, at least one suspicious lesionarea is characterized by increased blood activity (e.g., flow and/orvolume) in comparison to at least one area around normal tissue. In someembodiments, at least one natural anatomic characteristic of asuspicious lesion area (e.g., in soft or dense tissue), includes atleast one shadow and/or cloud-like object observed by at least onefluoroscopic image. In some embodiments, there are additional sourcesfor the at least one shadow and/or cloud-like object by at least onefluoroscopic image (e.g., at least one rib cross-section, joint, majorblood vessel, etc.)

In some embodiments, the present invention is a method that separates atleast two different (e.g., non-identical) portions of visible tissue(s)(which can be the same or different tissue) on a fluoroscopic image andcombines the at least two different portions into objects throughsegmentation and tracking of visible tissues using optical flow onfluoroscopic video. In some embodiments, the pixels on a fluoroscopicscreen are (1) classified by density range, (2) tracked through the livefluoroscopic video, and (3) classified by movement. For example,breathing includes lung expansion and contraction movements, which varyfrom lobe to lobe in the same lung and also vary from movement of ribs.Such movements result in a lung projection, and can be shown by thefluoroscopic video images generated from the inventive method of thepresent invention, characterized by a plurality (e.g., a variety) ofmovements for every distinguishable anatomical structure as illustratedby FIG. 6.

In some embodiments, the method of the present invention includes aregistering process/step, where the registering process/step uses asinput: a segmentation of bronchial airways from (i) a fluoroscopic imageand (ii) a CT scan. In some embodiments, a course and/or fineregistration is performed using a registering step.

In some embodiments, a method allows registration between at least onebronchial airway tree extracted from a preoperative CT image and airwaysextracted from fluoroscopic image sequence using augmented bronchogram.In an embodiment, a general flow is illustrated in FIG. 7.

In some embodiments, the present invention is an augmented bronchogram.In some embodiments, the augmented bronchogram is an augmented image ofinvisible airways (e.g., not visible by fluoroscopic image) and isextracted from fluoroscopic images.

In an embodiment, an augmented bronchogram is generated by injecting aradiopaque substance configured to make bronchi visible (FIG. 8). In anembodiment, visible bronchi provide information (1) to extract a partialbronchial tree from fluoroscopic images and (2) to register the partialbronchial tree to a second image, e.g., the bronchial tree extractedfrom a preoperative image. In some embodiments the radiopaque substanceinjected in bronchi does not highlight (i.e., make visible) the airwaysuniformly. In some embodiments, the radiopaque substance quicklydisappears from an image or disperses (e.g., but not limited to, within1-60 seconds, 1-45 seconds, 1-30 seconds, 1-15 seconds, etc.), whichdeteriorates fluoroscopic image quality (FIG. 9), and creates a blurredimage. In some embodiments of the present invention, at least one imageprocessing algorithm is utilized to generate a bronchogram. In someembodiments of the present invention, at least one image processingalgorithm is utilized to generate an augmented bronchogram.

In some embodiments, an augmented bronchogram is created by using atleast one radiopaque instrument, that has can optionally have anchoringmechanism as drawn by FIG. 14. In some embodiments, the radioscopicinstrument is visible in fluoroscopic images and represents ananatomical structure that can be registered to the bronchial tree, whichis identified from at least one preoperative image. In some embodiments,the direct extension of this method is using multiple instrumentpositions (FIG. 10) extracted and accumulated from temporal fluoroscopicimage sequence during the same procedure (FIG. 11). In some embodiments,the radiopaque instrument can be multi-lumen, where lumens can be usedfor: (i) diagnostic or treatment procedure, (ii) introducing multipleradiopaque guide-wires simultaneously into multiple bronchial airwaysand using the guide-wires as a plurality of registration references. Insome embodiments, this technique improves registration accuracy androbustness.

In some embodiments, an augmented bronchogram is created using at leastone instrument that allows perfusion of radiopaque substance to remainvisible and in place (e.g., substantially static) for an increasedperiod of time. In some embodiments, the increased period of time isachieved by using the at least one instrument that spreads at least oneradiopaque substance on the walls of airways using a brush or sprinkleson the tool exterior. In some embodiments, a radiopaque substance havinga high viscosity (e.g., in the form of hydrogel) is injected through theinstrument and dispersed on the airways. In some embodiments, theradiopaque material is configured to be gradually released from theradiopaque substance. In some embodiments, the airway area retains aradiopaque characteristic for longer period of time. In someembodiments, a reverse thermo-gelling polymer or similar material isused, to allow effective injection of liquid substance at a lowtemperature while prevention of fluoroscopic image quality deterioration(FIG. 9) or blurred fluoroscopic image since the injected substancebecomes a semisolid gel as the temperature increases to the bodytemperature.

In some embodiments, the present invention is a method that includesadding a third dimension (depth) to a position of an instrument on a 2Dfluoroscopic image. In some embodiments, a depth of at least one sectionof the instrument is calculated by (1) comparison of (a) the projectedinstrument shape on fluoroscopic image with (b) the known anatomicalstructure of the bronchial airway and (2) making an assumption ofconstrained instrument location inside the bronchial tree (FIG. 13).

In some embodiments, the present invention is a method that includesadding elevation of the instrument (orientation angle) in a directionperpendicular to a fluoroscopic image. In some embodiments, there are atleast two methods to calculate orientation magnitude: (1) comparing theprojected and actual physical lengths of a radiopaque straightinstrument section, which uses a known zoom (i.e., magnification) of thefluoroscopic image (e.g., from an available registration) (FIG. 12), and(2) using an orientation sensor attached to the instrument to calculatethe orientation of the instrument relative to the body of a patient orrelative to the fluoroscopic device.

In some embodiments, the method of the present invention includesintegrating information including 3D location and orientation todetermine the 6 degrees of freedom (DOF) of the instrument inside thepatient (e.g., a human body).

In some embodiments, the present invention is a method to track motionand orientation of a tip of an instrument using integrated sensorslocated on the tip. In some embodiments, the sensor is selected from agroup consisting of: a gyroscope, an accelerometer and/or amagnetometer. In some embodiments, the transmitted information fromthese sensors allows calculating the orientation and the location of thetip in real time. In some embodiments of the present invention, therobustness of the location calculation is improved (i.e., increasedaccuracy) by assuming/predicting the samples are inside the bronchi. Insome embodiments, the samples are registered to the 3D bronchial treeextracted from the preoperative CT image.

In an exemplary embodiment of the present invention, FIG. 7 is a flowchart illustrating method 700. In some embodiments, the flow chartpresents the registration process between bronchial airway treeextracted from preoperative image (e.g., but not limited to, a CTscan/image) and airways extracted from fluoroscopic images (e.g., 2, 3,4, 5, 6, 7, 8, 9, 10, etc.) using an augmented bronchogram. In someembodiments, 710 of the method 700, a CT and/or MRI is a sourcepreoperative and/or intraoperative image. In some embodiments, thepreoperative and/or intraoperative image is acquired and transformedinto 3D space, and used during surgical treatment and/or diagnosticprocedure for a treatment and/or a diagnosis. In an exemplaryembodiment, at 720 of the method 700, a 3D bronchial tree is extractedfrom the image 710 using (1) an automatic segmentation algorithm and/or(2) a manual notation by a physician. In an exemplary embodiment, at 705of the method 700, there is a source fluoroscopic image and/orfluoroscopic video captured from the fluoroscope. In an exemplaryembodiment, at 730 of the method 700, an augmented bronchogram iscalculated using fluoroscopic image 705 by one or more approachesdisclosed in the present invention.

In some embodiments, the method of the present invention includes anautomatic separation/segmentation between soft tissue, bones,instrument(s), an anatomical object(s), and background, where theautomatic separation/segmentation uses instrument and/or tissue movementto differentiate between different types of tissues/organs and/orinstruments (e.g., movement and/or density) to result in the generationof extracted information (e.g., a bronchial tree).

In an exemplary embodiment, the 3D bronchial tree extracted by 720 andaugmented bronchogram extracted by 730, are registered at 740 using themethod show in 700. In an exemplary embodiment, the registration processestimates pose information (e.g., position, orientation, and/or cameraparameters) of the fluoroscope that would project a 3D bronchial tree tomatch a 2D augmented bronchogram, and produces a correspondence between3D space of the image 710 and 2D space of the image 705.

In an embodiment, FIG. 8 shows a sample of augmented bronchogramobtained from a sequence of fluoroscopic images containing an injectedradiopaque substance that highlights a partial bronchial tree.

In an embodiment, FIG. 9 shows a fluoroscopic image, which is the samesubject as in FIG. 8, but the image was taken after 30 seconds ofinjection. As shown, the injected radiopaque substance diffuses to thesurrounding regions, producing a blurred image. In an embodiment, anaugmented bronchogram produces a clear image after 30 seconds ofinjection.

In an embodiment of the present invention, FIG. 10 shows an illustrationof the method of use of a radiopaque instrument that is visible onfluoroscopic images. In an embodiment, the images, e.g., 1005, 1010 and1015, show fluoroscopic views containing a visible instrument indifferent locations and a schematic structure of a bronchial tree thatis not visible in a real fluoroscopic image, and shown here forillustration purposes only. The instrument shown in views 1005, 1010 and1015 can be the same instrument or different instruments.

In an example, superposition of imaging incorporates distortioncorrection caused by body movement, breathing, instrument introduction,etc. In some embodiments, the temporal instrument positions are acquiredfor superposition at the predefined breathing phase.

In an exemplary embodiment, FIG. 11 illustrates the augmentedbronchogram, derived from the views 1005, 1010 and 1015 from FIG. 10. Inan embodiment, each view adds information regarding the surroundinganatomical structures. In an embodiment, the information is combined tocreate an augmented bronchogram.

In an embodiment, FIG. 12 shows a straight section of an instrument1205, located in the 3D space inside the body. In an embodiment, theinstrument is projected on the fluoroscope image plane 1210 and createdthe projection image 1215. In an embodiment, the angle between thestraight section of the instrument 1205 and the fluoroscope image plane1210 is “alpha.”

In an embodiment, FIG. 13 shows a 3D bronchial tree 1315, containing ananatomical path 1320, located inside the airways. In an embodiment, whenthe 3D anatomical path 1320 is projected on the fluoroscope image plane1315, the projection 1310 loses the original depth information. In anembodiment, the present invention recovers this information.

In an embodiment, FIG. 14 shows disposable navigation catheter withanchoring, that can be guided by means of pre-curved tip 1410 throughthe bronchial airways. The tool handle 1420 can be optionally used toenhance navigation performance. The catheter tip can be fixated insidethe bronchial airways by means of anchor 1440 that is designed asinflatable balloon or extendable spring, to allow instant multipleaccess to the area of interest around the catheter tip by medicalinstrument. The diagnostic and treatment instrument can be introducedthrough the working channel located inside the navigation catheter atthe entry point 1430.

In an embodiment, FIG. 15A shows a fluoroscopic image of the diagnosticprocedure in human lungs. Biopsy needle 1502 is protruding throughworking channel of the bronchoscope 1503 to biopsy the suspicious targetnodule, which is perceived by physician as dark region 1503. Theaugmented fluoroscopic image FIG. 15B is generated to highlight theactual nodule area 1504 that was marked by physician prior to procedureon correspondent preoperative CT image of patient chest. The augmentedimage preserves bronchoscope 1506 and needle 1505 at the originallocation, however the difference between actual 1506 and perceived 1503nodule position is obvious. The highlighting technique of 1506 isdemonstrated on FIG. 15B, where the yellow color is “injected” into thenodule area of the fluoroscopic image, which is correspondent to one ofthe CT image (and is further surrounded by a dashed line), while theoriginal information of fluoroscopic image is yet preserved.

In some embodiments, the instant invention is a method and flow thatallows using first imaging modality such as CT, MRI, etc., and planninginformation through generation of augmented image from second imagingmodality, such as fluoroscopy, digital subtraction angiography (DSA),etc., with highlighted area of interest or structures and optionallyadditional imaging and\or planning information, originated from a firstimaging modality, superimposed over it comprising: (i) using firstimaging modality to obtain at least one first image of chest; (ii)manual or automatic planning of procedure through defining landmarks,area of interest, incision points, critical structures, bifurcations,anatomical organs, etc.; (iii) acquire at least one-second image fromsecond imaging modality, such as fluoroscopy or DSA, and generation ofcompatible virtual image, such as DRR, from first imaging modality; (iv)mapping of planning data to the objects and structures on the compatiblevirtual image; (v) registration of at least one second image or videoframe from second imaging modality to first image or its portion sourcedfrom first imaging modality; (vi) transfer mapping (i.e., identifyingand mapping) of planning data from the compatible virtual image, sourcedfrom first imaging modality to second image from second imaging modalityby means of image registration; (vii) highlighting the area of interest,anatomical structures on second image sourced from second imagingmodality to obtain third image, wherein the third image is augmented.

In some embodiments, the method further includes superimposing of atleast one image or its derivative, it's portion or image based planninginformation sourced from first imaging modality over second imagingmodality. In some embodiments, the method further includes navigationand guidance instructions that aid movement of medical instrument. Insome embodiments, the method further includes guidance for positioningsecond imaging modality, such as fluoroscopic C-Arm, to allowmaintaining optimal visibility for the area of interest, incisionpoints, anatomical structures, tool access direction. In someembodiments, the method implements tracking of anatomic structures onsubsequent frames from second imaging modality, such as fluoroscopicvideo, having same acquisition parameters (mode, position, field ofview) to allow higher quality of augmented fluoroscopic image throughsuppression of static anatomic structures and improving signal to noiseof underlying soft tissue. In some embodiments, multiphase registrationis performed, where the static objects with small movement, such asribs, are registered at first and more dynamic objects such asdiaphragm, bronchi, blood vessels, etc. are gradually registered in thefollowing registration iterations. In some embodiments, the interferingstructures being deemphasized. In some embodiments, the compatiblevirtual image is not generated while the planning data from firstimaging modality is transferred to second imaging modality by means ofimage registration.

In some embodiments, the present invention is a method allowing for thegeneration of an augmented third image, such as intraoperativefluoroscopic, DSA, etc., with highlighted area of interest or structurescomprising: (i) using at least two intraoperative images with knownrelative movement and rotation to allow grouping pixels ofintraoperative image according to their movement variation and intensityvalues; (ii) performing registration or cross-correlation between atleast two sequential intraoperative images to reconstruct structures inthe area of interest; (iii) differentiating moving and static structuresin the area of interest on user demand; (iv) highlighting anatomicalstructures on intraoperative image, or any combination thereof. In someembodiments, the method includes using Chest X-ray radiographic image,while the said radiographic image serves as a reference image thatenables to enhance anatomical structures on second image throughregistration or cross-correlation of the information from radiographicimage.

In some embodiments, the present invention is an augmented fluoroscopydevice that allows generation of augmented fluoroscopy image comprising:a video and image processing unit; a video input card or externallyconnected device that is capable to input video signal from the varietyof Fluoroscopic device; a 3D planning input in internal or DICOM format;an augmented video signal output; or any combination thereof.

In some embodiments, the device is integrated within fluoroscopic deviceas a module, to obtain RAW data as a signal, and therefore having RAWdata input card instead of video input card. In some embodiments, thedevice is integrated within cone-beam CT system.

In some embodiments, the present invention is a tissue or anatomicalstructure highlighting technique, where the volume of interest isselected on the image sourcing from first imaging modality, such as CTor MRI; acquired image from second imaging modality; coarse registrationis performed between second and first imaging modalities to identify thepose of virtual camera in the second imaging modality correspondent tothe one of second imaging modality; at least one pattern is producedfrom first imaging modality for the anatomical structure around volumeof interest; the matching pattern is found in the second imagingmodality using single or multiple patterns produced from first imagingmodality; the matching pattern from the second imaging modality isenhanced to highlight the anatomy in the volume of interest producingthird imaging modality.

In some embodiments of the method of the present invention, when theanatomic structures located outside the area of interest are found andsuppressed using the same technique. In some embodiments, the pattern iscomprised from anatomical features such as airways, ribs, and bloodvessels. In some embodiments, the matching feature from second imagingmodality is derived from set of at least one instrument position insidethe area of interest.

A method of object depth calculation as follows: given the parameters ofcompatible virtual image sourcing from first imaging modality, such asDDR—to fluoroscopy; given the pose and field of view of virtual camera,such as virtual fluoroscopic camera, projecting first imaging modalityto second imaging modality; determine the object size on virtual image,such as ribs width on DDR at specific location; calculate the depth(such as distance of the specific object or object area fromfluoroscopic X-ray source) through comparison between the known objectsizes sourced from first image (e.g. CT image) to the one measured onsecond image (e.g. fluoroscopic image), or any combination thereof. Insome embodiments, object size is determined from technical specificationinstead of or in addition to the measurement on compatible virtualimage, such as tool rigid part length or width. In some embodiments, thecatheter-type tool is designed to allow the calculation of trajectory asa combination of depth distances from second imaging modality cameracenter.

A method and flow that allow registration of first three-dimensionalimaging modality such as CT, MRI, etc., with second two-dimensionalimaging modality of real time x-ray imaging such as fluoroscopy, digitalsubtraction angiography (DSA), etc. comprising: using first imagingmodality to obtain at least one first image of chest; perform manual orautomatic segmentation of natural body cavities such as bronchialairways in 3D space; acquire at least one images or sequence of videoframes from second imaging modality, such as fluoroscopy or DSA;generation of two-dimensional augmented image generated from secondimaging modality that combines unique information to describe the fullor partial map of natural body cavities such as portion of bronchialairway tree, abovementioned as augmented bronchogram; calculateregistration between first and second imaging modalities through poseestimation by fitting abovementioned corresponded features, or anycombination thereof. In some embodiments, an augmented bronchogram isgenerated using radiopaque material is injected to highlight the bodycavity.

In some embodiments, augmented bronchogram is generated throughsuperposition of imaging from at least two different temporal positionsof radiopaque instrument located inside the body cavity. In someembodiments, augmented bronchogram is generated through superposition ofimaging from at least one different positions of radiopaque instrumentlocated inside the body cavity and angular measurement of C-Armorientation relative to patient bed. In some embodiments, the radiopaqueinstrument is designed and configured to reconstruct itsthree-dimensional space from single projection. In some embodiments,radiopaque substances having a high viscosity such as, but not limitedto, hydrogel, reverse thermo-gelling polymer are used to generateaugmented bronchogram. In some embodiments, superposition of imagingincorporates distortion correction caused by body movement, breathing,instrument introduction etc. In some embodiments, the temporalinstrument positions are acquired for superposition at the predefinedbreathing phase. In some embodiments, the present invention is a devicefor navigating inside natural body cavity comprising: guided sheath withanchoring at the tip and guided wire. In some embodiments, the deviceincludes an inflatable balloon serving as anchoring mechanism.

In some embodiments, the instant invention provides a method, including:obtaining a first image from a first imaging modality; identifying onthe first image from the first imaging modality at least one element,where the at least one element comprises a landmark, an area ofinterest, an incision point, a bifurcation, an organ, or any combinationthereof, obtaining a second image from a second imaging modality;generating a compatible virtual image from the first image from thefirst imaging modality; mapping planning data on the compatible virtualimage; where mapped planning data corresponds to the at least oneelement, coarse registering of the second image from the second imagingmodality to the first image from the first imaging modality; identifyingat least one element of the mapped planning data from the compatiblevirtual image; identifying at least one corresponding element on thesecond imaging modality; mapping the at least one corresponding elementon the second imaging modality; fine registering of the second imagefrom the second imaging modality to the first image from the firstimaging modality; generating a third image; where the third image is anaugmented image including a highlighted area of interest.

In some embodiments, the method further includes superimposing the atleast one image, a portion of the at least one image, or a planninginformation derived from the first imaging modality over the secondimaging modality. In some embodiments, the method further includes usingat least one instruction, where the at least one instruction can includeinformation regarding navigation, guidance, or a combination thereof. Insome embodiments, the guidance includes information regarding apositioning of a device shown the second imaging modality, where thedevice comprises a fluoroscopic C-Arm, as to result in achievingvisibility for the area of interest, incision points, anatomicalstructures, or tool access direction. In some embodiments, the methodfurther includes tracking of at least one anatomical structure by use ofat least one subsequent image derived from the second imaging modality,where the second imaging modality comprises a fluoroscopic videoconfigured to have substantially the same acquisition parameters, andwhere the acquisition parameters comprise mode, position, field of view,or any combination thereof, to generate the augmented fluoroscopic imageby suppressing static anatomic structures and/or improving signal tonoise of underlying soft tissue. In some embodiments, the method furtherincludes performing a multiphase registration, where the at least onesubstantially static object is first registered; and where at least onedynamic object is second registered, where the at least one dynamicobject comprises a diaphragm, a bronchus, a blood vessel, or anycombination thereof. In some embodiments, the method further includesdeemphasizing at least one interfering structure. In some embodiments,the compatible virtual image is not generated while the planning datafrom first imaging modality is transferred to second imaging modality bymeans of image registration.

In some embodiments, the instant invention provides a method, including:

using at least two intraoperative images with known relative movementand rotation to generate a grouping of pixels derived from anintraoperative image, where the grouping of pixels is determined byindividual calculation of each pixel using: (a) movement variation ofeach pixel and (b) intensity values of each pixel; performingregistration using at least two sequential intraoperative images toreconstruct structures in an area of interest; differentiating movingstructures from static structures in the area of interest; andhighlighting anatomical structures on at least one intraoperative image.In some embodiments, the method further includes using a chest x-rayradiographic image as a first intraoperative image.

In some embodiments, the instant invention provides a system includingan augmented fluoroscopy device configured to generate an augmentedfluoroscopy image including (a) video and image processing unit, (b)video input card or externally connected device configured to inputvideo signal a fluoroscopic device, (c) 3D planning input in internal orDICOM format, (d) an augmented video signal output, or any combinationthereof. In some embodiments, the system is integrated with at least onefluoroscopic device is a module including a RAW data input card (i.e.,instead of a video input card) configured to obtain RAW data as asignal. In some embodiments, the system is integrated with a Cone-beamCT system.

In some embodiments, the instant invention provides a system includingan instrument for navigating inside natural body cavity including: (a) aguided sheath with anchoring at the tip and/or (b) a guided wire. Insome embodiments, the instrument is an inflatable balloon configured toact as an anchoring mechanism.

In some embodiments, the instant invention provides a method including:(i) selecting a volume of interest on a first image from a first imagingmodality; (ii) generating a second image from a second imaging modality;(iii) coarse registering using the first imaging modality and the secondimaging modality; (iv) producing at least one pattern from the firstimaging modality; (v) generating a matching pattern by use of the secondimaging modality using single or multiple patterns produced from firstimaging modality; (vi) enhancing the matching pattern from the secondimaging modality to highlight the anatomy in the volume of interest forproducing third imaging modality. In some embodiments, the anatomicstructures located outside the area of interest are found and suppressedusing substantially the same method. In some embodiments, the patternincludes anatomical features including, but not limited to, airways,ribs, and blood vessels. In some embodiments, the matching feature fromsecond imaging modality is derived from a set of at least one instrumentposition inside the area of interest.

In some embodiments, the instant invention provides a method including:using a first imaging modality to obtain at least one first image of apatient's chest; segmenting natural body cavities including bronchialairways in a 3D space; generating at least one image from a secondimaging modality; generating a two-dimensional augmented image generatedfrom the second imaging modality by combining information, where theinformation describes a complete map or a partial map of natural bodycavities, including a bronchial airway tree; calculating registrationbetween the first imaging modality and the second imaging modality aspose estimation between the portion of bronchial airway sourcing fromsecond imaging modality and segmented map of bronchial airway sourcingfrom first imaging modality; calculating registration between first andsecond imaging modalities through pose estimation by mappingcorresponding features. In some embodiments, the augmented bronchogramis generated using radiopaque material is injected to highlight the bodycavity. In some embodiments, the augmented bronchogram is generatedthrough superposition of imaging from at least three two differentpositions of radiopaque instrument located inside the body cavities. Insome embodiments, an augmented bronchogram is generated throughsuperposition of imaging from at least one different positions ofradiopaque instrument located inside the body cavity and angularmeasurement of C-Arm orientation relative to patient bed. In someembodiments, the radiopaque instrument is designed and configured toreconstruct its three-dimensional space from single projection. In someembodiments, the radiopaque substance(s) having a high viscosity suchas, but not limited to, hydrogel, reverse thermo-gelling polymer can beused to generate augmented bronchogram.

In some embodiments, the instant invention provides a method including:providing the parameters of compatable virtual image sourcing from firstimaging modality, such as, but not limited to, DDR—to fluoroscopy;determining an object size on virtual image, such as, but not limitedto, ribs width on DDR at specific location; providing the pose and fieldof view of a virtual camera, such as, but not limited to, a virtualfluoroscopic camera, projecting first imaging modality to second imagingmodality such as fluoroscopic camera calculated from calibrationprocess; determining the object size on the virtual image, such as ribswidth on DDR at specific location; calculating the depth (for example,but not limited to, distance of the specific object or object area fromfluoroscopic X-ray source) through comparison between the known objectsizes sourced from first image (e.g. CT image) to the one measured onsecond image (e.g. fluoroscopic image). In some embodiments, the objectsize is determined from technical specification instead of or inaddition to the measurement on compatible virtual image, such as toolrigid part length or width. In some embodiments, the catheter-type toolis designed to allow the calculation of trajectory as a combination ofdepth distances from second imaging modality camera center.

While a number of embodiments of the present invention have beendescribed, it is understood that these embodiments are illustrativeonly, and not restrictive, and that many modifications may becomeapparent to those of ordinary skill in the art. Further still, thevarious steps may be carried out in any desired order (and any desiredsteps may be added and/or any desired steps may be eliminated).

Although the invention has been described in terms of particularembodiments and applications, one of ordinary skill in the art, in lightof this teaching, can generate additional embodiments and modificationswithout departing from the spirit of or exceeding the scope of theclaimed invention. Accordingly, it is to be understood that the drawingsand descriptions herein are proffered by way of example to facilitatecomprehension of the invention and should not be construed to limit thescope thereof.

What is claimed is:
 1. A method, comprising: obtaining a first imagefrom a first imaging modality; identifying on the first image from thefirst imaging modality at least one element, wherein the at least oneelement comprises a landmark, an area of interest, an incision point, abifurcation, an organ, or any combination thereof; obtaining a secondimage from a second imaging modality; generating an augmentedbronchogram image from the second imaging modality, wherein theaugmented bronchogram image is generated by: placing a radiopaqueinstrument inside a body cavity at a first position, generating a firstimage of the radiopaque instrument inside the body cavity at the firstposition, placing the radiopaque instrument inside the body cavity at asecond position, generating a second image of the radiopaque instrumentinside the body cavity at the second position, superpositioning thefirst image of the radiopaque instrument inside the body cavity at thefirst position with the second image of the radiopaque instrument insidethe body cavity at the second position to generate the augmentedbronchogram; wherein the first position and the second position aredifferent; wherein the augmented bronchogram is a map of a bronchialairway tree; generating a compatible virtual image from the first imagefrom the first imaging modality; coarse registering of the augmentedbronchogram from the second imaging modality to the compatible virtualimage from the first imaging modality; fine registering of the augmentedbronchogram image from the second imaging modality to the compatiblevirtual image from the first imaging modality; generating a third image;wherein the third image is an augmented image including a highlightedarea of interest, wherein the highlighted area of interest can bedetermined by a method selected from the group consisting of: (i)bolding a portion of the augmented bronchogram image, (ii) coloring theportion of the augmented bronchogram image, (iii) enhancing the portionof the augmented bronchogram image, (iv) super-positioning a graphicover the augmented bronchogram image, (v) or any combination thereof. 2.The method of claim 1, further comprising superimposing the first image,a portion of the first image, or a first planning data derived from thefirst imaging modality over the second image of the second imagingmodality.
 3. The method of claim 1, further comprising using at leastone instruction, wherein the at least one instruction can includeinformation regarding navigation, guidance, or a combination thereof. 4.The method of claim 3, wherein the guidance includes informationregarding the first position of the radiopaque instrument shown by thesecond imaging modality, wherein the radiopaque instrument comprises afluoroscopic C-Arm, to result in achieving visibility for the area ofinterest, incision points, anatomical structures, or tool accessdirection.
 5. The method of claim 1, further comprising tracking of atleast one anatomical structure by use of at least one subsequent imagederived from the second imaging modality, wherein the second imagingmodality comprises a fluoroscopic video configured to have substantiallythe same acquisition parameters, and wherein the acquisition parameterscomprise mode, position, field of view, or any combination thereof, togenerate the augmented fluoroscopic image by suppressing static anatomicstructures, improving signal to noise of underlying soft tissue, or anycombination thereof.
 6. The method of claim 1, further comprisingperforming a multiphase registration, wherein the at least onesubstantially static object is first registered; and wherein at leastone dynamic object is second registered, wherein the at least onedynamic object comprises a diaphragm, a bronchus, a blood vessel, or anycombination thereof.
 7. The method of claim 1, further comprisingdeemphasizing at least one interfering structure.
 8. The method of claim2, wherein the compatible virtual image is not generated while theplanning data from first imaging modality is transferred to secondimaging modality by means of image registration.
 9. The method of claim1, wherein the augmented bronchogram is a partial map of the bronchialairway tree.
 10. The method of claim 1, wherein the augmentedbronchogram is a complete map of the bronchial airway tree.
 11. Themethod of claim 1, wherein the augmented bronchrogram image is furthergenerated by a method selected from the group consisting of: a)superpositioning the first position of the radiopaque instrument, thesecond position of the radiopaque instrument in the body cavity, andtheir correspondent angular measurement of a C-arm orientation relativeto the patient, or any combination thereof; b) using a radiopaqueinstrument configured to reconstruct a three-dimensional space from asingle projection; b) or any combination thereof.